Getting started#

The xdatasets library enables users to effortlessly access a vast collection of earth observation datasets that are compatible with xarray formats.

The library adopts an opinionated approach to data querying and caters to the specific needs of certain user groups, such as hydrologists, climate scientists, and engineers. One of the functionalities of xdatasets is the ability to extract data at a specific location or within a designated region, such as a watershed or municipality, while also enabling spatial and temporal operations.

To use xdatasets, users must employ a query. For instance, a straightforward query to extract the variables t2m (2m temperature) and tp (Total precipitation) from the era5_reanalysis_single_levels dataset at two geographical positions (Montreal and Toronto) could be as follows:

query = {
    "datasets": {"era5_reanalysis_single_levels_dev": {'variables': ["t2m", "tp"]}},
    "space": {
        "clip": "point", # bbox, point or polygon
        "geometry": {'Montreal' : (45.508888, -73.561668),
                     'Toronto' : (43.651070, -79.347015)
                    }
    }
}

An example of a more complex query would look like the one below.

Note

Don’t worry! Below, you’ll find additional examples that will assist in understanding each parameter in the query, as well as the possible combinations.

This query calls the same variables as above. However, instead of specifying geographical positions, a GeoPandas.DataFrame is used to provide features (such as shapefiles or geojson) for extracting data within each of them. Each polygon is identified using the unique identifier Station, and a spatial average is computed within each one (aggregation: True). The dataset, initially at an hourly time step, is converted into a daily time step while applying one or more temporal aggregations for each variable as prescribed in the query. xdatasets ultimately returns the dataset for the specified date range and time zone.

query = {
    "datasets": {"era5_reanalysis_single_levels_dev": {'variables': ["t2m", "tp"]}},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "aggregation": True, # spatial average of the variables within each polygon
        "geometry": gdf,
        "unique_id": "Station" # unique column name in geodataframe
    },
    "time": {
        "timestep": "D",
        "aggregation": {"tp": np.nansum,
                        "t2m": [np.nanmax, np.nanmin]},

        "start": '2000-01-01',
        "end": '2020-05-31',
        "timezone": 'America/Montreal',
    },
}

Query climate datasets#

In order to use xdatasets, you must import at least xdatasets, pandas, geopandas, and numpy. Additionally, we import pathlib to interact with files.

[1]:

%load_ext autoreload
%autoreload 2

[2]:

import xdatasets as xd
import geopandas as gpd
import pandas as pd
import numpy as np

# Visualization
import hvplot.xarray
import panel as pn

from pathlib import Path

/home/runner/work/xdatasets/xdatasets/xdatasets/core.py:6: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:

import os
os.environ['USE_PYGEOS'] = '0'
import geopandas

In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
  import geopandas as gpd

Clip on polygons with no averaging in space#

Let’s first access certain polygon features, which can be in the form of shapefiles, geojson, or any other format compatible with geopandas. In this example, we are using JSON (geojson) files.

[8]:

bucket = Path('https://s3.us-east-2.wasabisys.com/watersheds-polygons/MELCC/json')

paths = [bucket.joinpath('023003/023003.json'),
         bucket.joinpath('031101/031101.json'),
         bucket.joinpath('040111/040111.json')]

Subsequently, all of the files can be opened and consolidated into a single geopandas.GeoDataFrame object.

[9]:

gdf = pd.concat([gpd.read_file(path) for path in paths]).reset_index(drop=True)
gdf

[9]:

	Station	Superficie	geometry
0	023003	208.4591919813271	POLYGON ((-70.82601 46.81658, -70.82728 46.815...
1	031101	111.7131058782722	POLYGON ((-73.98519 45.21072, -73.98795 45.209...
2	040111	433.440893903503	POLYGON ((-74.06645 46.02253, -74.06647 46.022...

Let’s examine the geographic locations of the polygon features.

[10]:

gdf.hvplot(geo=True,
           tiles='ESRI',
           color='Station',
           alpha=0.8,
           width=750,
           height=450,
           legend='top',
           hover_cols=['Station','Superficie'])

[10]:

The following query seeks the variables t2m and tp from the era5_reanalysis_single_levels dataset, covering the period between January 1, 1959, and September 30, 1961, for the three polygons mentioned earlier. It is important to note that as aggregation is set to False, no spatial averaging will be conducted, and a mask (raster) will be returned for each polygon.

[11]:

query = {
    "datasets": {"era5_reanalysis_single_levels": {'variables': ["t2m", "tp"]}},
    "space": {
        "clip": "polygon", # bbox, point or polygon
        "averaging": False, # spatial average of the variables within each polygon
        "geometry": gdf,
        "unique_id": "Station" # unique column name in geodataframe
    },
    "time": {
        "start": '1959-01-01',
        "end": '1963-08-31',
    },
}

xds = xd.Query(**query)

Spatial operations: processing polygon 040111 with era5_reanalysis_single_levels: : 3it [00:00,  4.75it/s]

By accessing the data attribute, you can view the data obtained from the query. For each variable, the dimensions of time, latitude, longitude, and Station (the unique ID) are included. In addition, there is another variable called weights that is returned. This variable specifies the weight that should be assigned to each pixel if spatial averaging is conducted over a mask (polygon).

[12]:

xds.data

Weights are much easier to comprehend visually, so let’s examine the weights returned for the station 023003. Notice that when selecting a single feature (Station 023003 in this case), the shape of our spatial dimensions is reduced to a 2x2 pixel area (longitude x latitude) that encompasses the entire feature.

[13]:

station = '023003'

ds_station = xds.data.sel(Station=station)
ds_clipped = xds.bbox_clip(ds_station).squeeze()
ds_clipped

[14]:

(
    (
        ds_clipped.t2m.isel(time=0).hvplot(
            title="The 2m temperature for pixels that intersect with the polygon on January 1, 1959",
            tiles="ESRI",
            geo=True,
            alpha=0.6,
            colormap="isolum",
            width=750,
            height=450,
        )
        * gdf[gdf.Station == station].hvplot(
            geo=True,
            width=750,
            height=450,
            legend="top",
            hover_cols=["Station", "Superficie"],
        )
    )
    + ds_clipped.weights.hvplot(
        title="The weights that should be assigned to each pixel when performing spatial averaging",
        tiles="ESRI",
        alpha=0.6,
        colormap="isolum",
        geo=True,
        width=750,
        height=450,
    )
    * gdf[gdf.Station == station].hvplot(
        geo=True,
        width=750,
        height=450,
        legend="top",
        hover_cols=["Station", "Superficie"],
    )
).cols(1)

[14]:

The two plots depicted above show the 2m temperature for each pixel that intersects with the polygon from Station 023003 and the corresponding weights to be applied to each pixel. In the lower plot, it is apparent that the majority of the polygon is situated in the upper-left pixel, which results in that pixel having a weight of approximately 91%. It is evident that the two lower pixels have very minimal intersection with the polygon, which results in their respective weights being nearly zero (hover on the plot to verify the weights).

In various libraries, either all pixels that intersect with the geometries are kept, or only pixels with centers within the polygon are retained. However, as shown in the previous example, utilizing such methods can introduce significant biases in the final outcome. Indeed, keeping all four pixels intersecting with the polygon with equal weights when the temperature values in the lower pixels are roughly 2 degrees lower than that of the upper-left pixel would introduce significant biases. Therefore, utilizing weights is a more precise approach.

Query hydrological datasets#

Hydrological queries are still being tested and output format is likely to change. Stay tuned!

[23]:

%%time
query = {
    "datasets": {'hydrometric': {'variables': ['streamflow',
                                               't2m_nanmax',
                                               't2m_nanmin',
                                               'tp_nansum'],
                                 'id': ['01010070', '01016500', '01017290','02*'],
                                }
                },
}
xds = xd.Query(**query)
xds.data

CPU times: user 10.3 s, sys: 886 ms, total: 11.2 s
Wall time: 12.9 s

[23]:

<xarray.Dataset>
Dimensions:     (id: 504, time: 44990, source: 1)
Coordinates:
  * id          (id) object '01010070' '01016500' ... '02492343' '02492360'
  * time        (time) datetime64[ns] 1900-01-01 1900-01-02 ... 2023-03-06
  * source      (source) <U11 'hydrometric'
Data variables:
    streamflow  (id, time, source) float64 dask.array<chunksize=(1, 22495, 1), meta=np.ndarray>
    t2m_nanmax  (id, time, source) float32 dask.array<chunksize=(1, 44990, 1), meta=np.ndarray>
    t2m_nanmin  (id, time, source) float32 dask.array<chunksize=(1, 44990, 1), meta=np.ndarray>
    tp_nansum   (id, time, source) float32 dask.array<chunksize=(1, 44990, 1), meta=np.ndarray>

[24]:

id1 = pn.widgets.Select(value="01010070", options=list(xds.data.id.values), name="id")
source = pn.widgets.Select(
    value="hydrometric", options=list(xds.data.source.values), name="source"
)


@pn.depends(id1, source)
def plot_hydrograph_and_weather(id1, source):
    da = xds.data.sel(id=id1, source=source)
    dx = da["streamflow"].dropna("time", how="any")

    trace1 = da["streamflow"].hvplot(
        grid=True,
        widget_location="bottom",
        color="black",
        xlim=(dx.time[0].values, dx.time[-1].values),
        title=f"Daily streamflow at location {id1}",
        width=850,
        height=300,
    )
    trace2 = da[["t2m_nanmax", "t2m_nanmin"]].hvplot(
        grid=True,
        widget_location="bottom",
        color=["red", "blue"],
        xlim=(dx.time[0].values, dx.time[-1].values),
        title=f"Daily minimum and maximum temperature at location {id1}",
        width=850,
        height=300,
    )

    trace3 = da[["tp_nansum"]].hvplot(
        grid=True,
        color=["turquoise"],
        xlim=(dx.time[0].values, dx.time[-1].values),
        title=f"Daily precipitation at location {id1}",
        width=850,
        height=300,
    )

    return pn.Column(trace1, trace2, trace3)


pn.Row(
    pn.Column("## Hydrometric Data Explorer", id1, source, plot_hydrograph_and_weather)
)

[24]:

Getting started#

Query climate datasets#

Clip by points (sites)#

Clip on polygons with no averaging in space#

Clip on polygons with averaging in space#

Bounding box (bbox) around polygons#

Query hydrological datasets#